Repository navigation
fix(FIX-2166): a held notice reaches its filer soon after a crash ends the filing request - #3834
Conversation
…s the filing request A notice held for its running filing request (FIX-2123) waited out its resend backoff when that request's server died mid-run: the request still read in_progress until the stale-request sweep marked it interrupted, and nothing then sent the held notice before its next resend. - waits.running answers false for an in_progress request whose registry heartbeat is silent past the host's stale-request threshold, where the flow's heartbeat is on and beats at least twice per threshold; otherwise the record alone answers, as before. - The stale-request sweep and both startup sweeps bring the wakes kept for an interrupted request's session forward to now, so the filer's pass sends what it held at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
jhoffner
left a comment
There was a problem hiding this comment.
Stand-in review (Bugbot is out of usage) of head 57323b9.
Verdict. The src change is sound. The multi-process safety, wake versioning and per-entry early-fire checks all came out clean (details inline). There is one 🔴, and it is in the new e2e test: a status assertion that races the sweep and failed 4 of 26 cases locally.
What I ran (in a throwaway worktree):
notice-after-crash.test.ts13× at head: 22/26 cases passed. The 4 failures were allexpected 'in_progress' to be 'interrupted', and in every one of them arm 1 had delivered the notice at ~7,050 ms, before the sweep's next tick.- All engine/core src reverted to the merge base: red at the upper bound (14,061 / 14,073 ms against 10,500 ms). The test catches the bug.
- Only
wait-chain.tsreverted: green, ~8,080 ms in every run. The e2e pins arm 2 (bring the wakes forward). Arm 1 (the heartbeat read) is proven only by the unit test. - The engine unit tests (
wait-chain,request-recovery,liveness-gate): 43/43 pass.
Findings
- 🔴
notice-after-crash.test.ts:145: theinterruptedstatus assertion races the sweep when arm 1 wins. Wait for the status, or drop the line. - 🟡
wait-chain.ts: the threshold is per process, not per deployment, and the stopped read stays on when sweeping is disabled. Make both deliberate in the JSDoc. Double delivery is ruled out bytakeNoticeandended, which still reads the record only. The remaining cost is the notice running beside a live filer. - 🟡
request-wakes.ts: a pass that was already in flight can settle over the brought-forward wake, becausemovedoesn't bump the version. That falls back to the backoff, so it isn't a regression. FIX-2150's re-owe is safe. Every entry pays only what is due, andwaitDeadlineis never moved. - 🟡
request-recovery.ts: optionallastHeartbeatAt == nullguard. Also a minor divergence from the sweep on records that aren't org-attributed. - 🟡 test teardown:
second.dispose()spends ~10 s draining in every case. There is also a glued comment on line 140.
Restraint and changeset. The engine patch is right: it adds an optional wakeEntriesOf on exported functions, pre-1.0, and the core change is JSDoc only. The README paragraph in orchestration was spliced into the middle of an existing line, which leaves one very long line; re-wrap it.
Generated by Claude Code
…rward, fail-safe heartbeat read - notice-after-crash: wait for the sweep's interrupted mark instead of racing it; add a sweep-off case that pins the heartbeat read on its own; short router drain at teardown. - bringSessionWakesForward owes from the wake it read (min due, version bump), so a pass in flight settles stale instead of undoing it. - waits.running treats an entry with no heartbeat as alive; JSDoc says the threshold is per process and the read stays on with sweeping off. - orchestration README paragraph re-wrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
A run left hanging holds the process's request-end drain, which every later host's shutdown in the same test worker waited on, timing out afterEach hooks in other files. The never-answering model call now answers the teardown's abort, and the teardown checks the run ended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87ce26b5bf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
waits.running read a live in_progress request as stopped when the active-request registry lookup threw: the rejection reached the outer catch, which answers false. Its notice was then sent, and its lapsed slots released, while it still ran. The heartbeat lookup now fails safe to the record's answer, as a missing entry or heartbeat does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
The problem
If the server dies while a worker's filing request is still running, a task ending that request filed does not reach the worker for a long time. After the restart the request still reads
in_progress. Each delivery of the notice is turned away for it ({ owed: false },noticeDeferred … whileRunning). The stale-request sweep marks it interrupted at least 60 s later. Then the notice waits again for its next resend, up to 5 minutes later. The FIX-2087 closure saw this in leg b in 1 of 6 runs.The delay was bounded, not a loss. With the defaults, it was about 90 s plus one resend backoff.
What this does
waits.runninganswersfalsefor anin_progressrequest whose registry heartbeat has been silent past the host's stale-request threshold. This is the same threshold the sweep uses, so a request on a live peer (a BullMQ worker) still reads as running. Some flows can't tell from the heartbeat: heartbeats off (heartbeatIntervalMs: 0), or beating slower than half the threshold. Those keep today's answer. An entry with no heartbeat recorded also keeps today's answer.With the defaults, a held notice is heard 60 to 90 s after the request's last heartbeat, plus about 2 s. Because a request beats every 10 s, that is about 50 to 92 s after the kill.
What's asked of you
Nothing here is yours to decide. The owner accepted a bound of about a minute for now. The owner-liveness lease that would make this a few seconds is filed as a separate follow-up.
Parts worth reviewing closely
packages/engine/src/execution/request-recovery.ts+durability/request-wakes.tsbringSessionWakesForward: the engine doesn't know which internal entry is the filer's pass, so it reads one wake for each internal entry the session's flow declares. For each wake kept there that is due later, it writes oneowe, built from that wake. Theowekeeps the earlier due time and bumps the version, so a pass in flight settlesstalerather than undoing the bring-forward. A wake whose entry is not the pass fires early. Each entry pays only what is due, then moves its wake on, so firing early costs one extra pass and never acts early.packages/engine/src/durability/wait-chain.tsrunning: a request with no registry entry still answers from its record. That is no proof its process is gone.takeNoticeandwaits.endedgate the take. The sweep would reap that request under the same misreading.packages/engine/src/context/liveness-gate.tsheartbeatStaleThreshold: the same rule as the gate's second arm. It reads a registry that isn't shared across processes too. It deliberately ignores the third arm: with sweeping off, this read is the only thing that frees the notice.Links
Linear FIX-2166. Closure: #3692 (FIX-2087, epic FIX-2056). Merges to
mainonce reviewed.How to review this
How to review this. This is a bug fix on the direct route: there is no spec and no spec PR, because the hard part was the diagnosis and it happened in code. This PR is the only gate, so unlike a spec-backed change, the approach is in scope: nothing here was signed off upstream.
Most valuable here: the diagnosis chain. Does the repro actually reproduce the reported failure? Is the named cause the real one? Does the fix address that cause rather than the symptom? Does the regression test fail without the fix? Then the second path: the legacy shape, the null boundary, the concurrent case and the cancel path.
In scope to challenge: where the guard was placed, any behaviour change a user could notice, and whether this should have been a spec'd feature instead of a fix.
Engineering calls
RequestHostorWakeStoremethod.detectInterruptedRequestsandcreateStaleRequestSweepergain one optional option,wakeEntriesOf. The shipped sweeps fill it from the flow registry: the router's stale sweep,FlowState's startup sweep, and the route handlers' startup sweep.busyrefusal for a request that is parking untouched.requestStates. A crashed request whose heartbeat is stale readsparked(not running, not ended). So its held notice and its lapsed wait slots are released then. A notice it took still waits for the sweep'sinterruptedmark before another copy takes it over, as before.check-interruptedroute does not bring wakes forward. It is a client-poked sweep. The periodic sweep that follows covers it.@flow-state-dev/core: JSDoc only, no changeset.@flow-state-dev/engine: patch changeset.Everything else: evidence
Diagnosis. The closure's kept store (
b-1-…sqlite) shows the following:workrequest isin_progress, and its registry entry never beat (killed 2.3 s in; the heartbeat interval is 10 s).onTaskSettledrequests returned{ owed: false }, and the wake in writer's session was still due.A throwaway check on main showed the notice heard about 8 s after the request was marked interrupted, at the next resend: bounded, not lost.
Red then green.
packages/orchestration/test/tasks/notice-after-crash.test.tsruns on in-memory stores and on SQLite:Case 1 (the sweep on):
interrupted.expected 14059 to be less than or equal to 9500(in-memory) and14057(SQLite). That is the backoff resend.Case 2 (the sweep off):
in_progress.Both stores, 4 cases per run: 20 runs, 80/80 cases passed, both before and after merging main (FIX-2170).
Unit tests:
engine/test/wait-chain.test.ts: stale, fresh, at-threshold, queued, no-heartbeat and unregistered entries; heartbeats-off fallback.engine/test/request-recovery.test.ts:engine/test/context/liveness-gate.test.ts:heartbeatStaleThreshold.Checks run, after merging main:
pnpm typecheck,pnpm lintandpnpm fallow:ci.node scripts/check-retired-terms.mjs.node goals/vocabulary/one-word-per-thing/sweep.mjs.Generated by Claude Code